feat(core): Support schema management over REST in RESTCatalog - #9673
Conversation
Add a Catalog-level API for listing schemas and expose it over the REST
protocol so that RESTCatalog-backed tables can read historical schemas
without direct filesystem access.
* Introduce `Catalog#supportsSchemaManagement` and
`Catalog#listSchemas(Identifier, SchemaFilter)` with a
`SchemaFilter` value object (all / latest / earliest / by id / by
range).
* Add `GET /v1/{prefix}/databases/{db}/tables/{obj}/schemas` with a
`ListSchemaResponse` payload; encode `SchemaFilter` as query
parameters (`latest`, `earliest`, `schemaId`, `maxSchemaId`,
`minSchemaId`).
* Implement `RESTCatalog#supportsSchemaManagement`/`listSchemas` and
map REST errors to catalog exceptions.
* Add `CatalogSchemaManager`, a `SchemaManager` that delegates to the
owning `Catalog` (analogous to `CatalogBranchManager`); reads go
through `listSchemas`, writes reuse existing
`Catalog#createTable`/`alterTable`/`rollbackSchema`.
* Wire `AbstractFileStoreTable#schemaManager` to prefer
`CatalogSchemaManager` whenever `supportsSchemaManagement()` is true.
* Extend the REST mock server and add tests covering the new filter
variants and the catalog-backed schema manager.
Co-authored-by: TRAE CLI <traecli@bytedance.com>
|
Hi @JingsongLi , this PR adds schema management support to RESTCatalog. Could you please take a look when you have time? Thanks! |
JingsongLi
left a comment
There was a problem hiding this comment.
- API design
Could we follow the existing loadSnapshot and listSnapshotsPaged pattern here, with loadSchema(identifier, version) and listSchemasPaged(identifier, maxResults, pageToken)? The version could support LATEST, EARLIEST, and a schema ID. This would keep the API consistent and avoid introducing SchemaFilter. The responses should carry the complete TableSchema to preserve its metadata.
- Backward compatibility
These new APIs should only serve schema display use cases. Existing functionality must continue using its current implementation so that newer clients remain compatible with older REST servers.
Could we remove CatalogSchemaManager and the related capability flag and table-environment wiring from this PR? Please also add a compatibility test against a server without the new endpoints, verifying that existing operations still work and never call those endpoints.
Purpose
Follow up on #9174 by adding display-only schema history APIs to
RESTCatalog.The APIs follow the existing snapshot API pattern and allow clients to inspect complete historical
TableSchemametadata through the REST service. Existing table operations and schema-management internals remain unchanged.API
Catalog
loadSchemasupports:LATESTEARLIESTlistSchemasPagedreturns schemas in descending schema ID order.REST protocol
The responses contain complete
TableSchemaobjects so that schema metadata is preserved.Backward compatibility
getTable,alterTable, androllbackSchemabehavior continues to use the existing implementation.SchemaManager, orCatalogEnvironmentwiring is introduced.Tests
Added coverage for:
LATEST,EARLIEST, and a specific schema IDValidation performed:
All targeted tests passed: 4 tests, 0 failures, 0 errors.